Papers with voice activity projection
Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Backchanneling is a form of feedback that is produced by listeners in a conversation . since the advent of ChatGPT, modern dialogue systems exhibit answer quality levels on par with humans in various professions. |
| Approach: | They propose a multi-task learning approach that learns textual representations for the task of backchannel prediction in tandem with dialogue act classification. |
| Outcome: | The proposed approach improves the prediction of specific backchannels by up to 2.0% in F1 . the audio encoder is pre-trained in a self-supervised fashion using voice activity projection . |
Multilingual Turn-taking Prediction Using Voice Activity Projection (2024.lrec-main)
Copied to clipboard
| Challenge: | a monolingual model does not make good predictions when applied to other languages, but a multilingual model is able to discern the language of the input signal. |
| Approach: | They propose to use a multilingual voice activity projection model to predict voice activities of spoken dialogue participants in English, Mandarin, and Japanese data. |
| Outcome: | The proposed model predicts the upcoming voice activities of participants in dyadic dialogue on multilingual data, encompassing English, Mandarin, and Japanese. |